posted 06-01-2004 12:05 PM            

In a nut shell, the individual known to us as ‘wdc’ is claiming that they are a truthful person deemed deceptive/significant responder (false positive) because of educating themselves about polygraph prior to the exam.

Lets discuss our thoughts and the research on this topic.

posted 06-01-2004 01:01 PM            

An Information Countermeasures Has No Effect on the Validity of the Test for
Espionage and Sabotage (TES)

Wendy R. Alloway and Charles R. Honts
Boise State University

The Test for Espionage and Sabotage (TES) is a relatively new polygraph test that was developed by the Department of Defense. No independent research has been done on the validity of that test. Moreover, WWW pages have recently appeared that purport t offer information on how the beat the TES through the use of countermeasures. In particular, The Lie Behind The Lie Detector, a book discussing polygraph countermeasures, was recently published on the internet. The stated purpose for the countermeasures information contained in the book is to help the innocent to pass their polygraph exams. However, it is clear that guilty people could also access this information and possibly use it to their advantage. The current study investigated the basic validity of the TES and the effects of providing readily available countermeasures information on the validity of the TES.

The participants in the study were forty undergraduate introductory psychology students who participated for course credit. The students were randomly assigned to one of four conditions in a 2 X 2 factorial experiment: Innocent Control, Innocent Countermeasures, Guilty Control, and Guilty Countermeasures. The guilty participants were instructed to steal an envelope from an office door. The envelope contained a voucher for two free movie passes. Innocent participants were instructed to drop off an envelope in a mail slot. The participants who were in a countermeasures condition were given a copy of The Lie Behind the Lie Detector to take home and read. All of the participants were given a polygraph after at least a week had passed from their initial appointment. The TES was administered on a computerized system according to the standards provided by the United States government. The participants were informed that they would receive a bonus in the form of two movie passes if they were able to pass their polygraph test. The physiological data were scored with the discriminant analysis model developed by Kircher and Raskin (1988). That model results in an a posteriori probability of truthfulness. Those probabilities were used as the dependent measure in this study.
The probabilities of truthfulness were analyzed with a 2 X 2 factorial ANOVA.
The factors were Guilt (Innocent or Guilty of the mock crime) and Countermeasure (did
or did not read the countermeasures material). ANOVA revealed a main effect of Guilt,
F (1,33) = 5.90, p = 0.02. The mean probability of truthfulness for Guilty subjects was
.418 (SD= .33), and the mean probability of truthfulness for Innocent subjects was .70
(SD=.33). Neither the main effect of Countermeasure nor the interaction of Guilt and Countermeasure were significant.

The TES produced a significant difference between Guilt and Innocence participants, providing support for the validity of the TES. Exposure to The Lie behind the Lie Detector, failed to produce any effects on the probabilities of truthfulness generated by either innocent or guilty participants. These results replicate previous research (Rovner, et al., 1978) that found reading countermeasures information had no effect on the validity of the polygraph. These findings provide some support for the continued use of the TES as a screening tool.

posted 06-01-2004 01:12 PM            

The simple truth is that in the testing we do research shows known error rates and we can't get away from them. We can only try to eliminate the ones caused by us doing a poor job.

As for knowing how a test works causing false positives, I disagree.
Could a subject respond to a question because he is thinking, "This is a relevant question,I can't respond to this question"? Why would a truthful person be thinking that anyway?

I believe someone already sited research by Raskin and Honts relating to this issue. I believe they determined that knowledge of testing procedures had little effect on the outcome of the test.

Unfortunately wdc, if he is what he claims to be,is just frustrated and looking for help. If I were in the same position I would be frustrated also and searching for help.

Jack

posted 06-01-2004 01:36 PM            

Is their any reason that we should NOT go to a 'directed lie' method?

I know the answer is probably not simplistic, but still it seems we need to be moving toward some different approach for the long term health of the profession, do you agree?

------------------
Ralph Hilliard
PolygraphPlace Owner & Operator
http://www.polygraphplace.com

posted 06-01-2004 03:06 PM            

how about someone researching a 5 chart 2 or 3 relevant question, R and I protocol.

I'll bet that would have some validity where significance and consistancy were present...

Jim

posted 06-01-2004 11:38 PM            

If we are to subscribe to the notion that psychological set is the underlying concept that draws one to respond to a certain question, then a subject must be focused on the most prevalent threat to them. When the person has the knowledge that they will not fail when they lie to the comparison questions and/or are directed to lie to the comparison questions it would seem logical that the threat would be diminished, in some cases maybe even lost. Although an innocent person theoretically will have a diminished or non- response to the relevant questions, there still are some inherent threats of the question to the innocent. Now take the information that ‘Good ole George’ gives to the innocent in the form of use countermeasures even if you are innocent or you will respond and fail absent their use. This information could hypothetically increase the threat of the relevant questions for the innocent. Now we have a flip of psychological set based on knowledge of the psychophysiological testing procedure and misinformation.

As for research, I for one do not believe that Honts’ study adequately answers such a question. First, Honts’ study follows the general reproduction of past polygraph laboratory research study procedures. The study also, as with some previous studies, attempts to answer too many questions with a very small sample size. Second, the information provided here shows no evidence of pre-study and post- study measurement for knowledge of the polygraph testing procedure. This method can help track whether or not those subjects in the given countermeasure pools, or any pool, had prior knowledge of the procedure, did follow the instructions as to read the LBLD material, and their final level of knowledge post polygraph exam. Finally, it does not address whether or not the polygraph information had an adverse effect on truthful people being deemed deceptive. Mainly because without a measurement for post-test knowledge of the procedure and information on whether or not the subjects chose to employ any countermeasures, and if so what, there is no meaningful conclusion that can be drawn. For instance, if the pool of 10 innocent countermeasures examinees all employed countermeasures and all were deemed truthful and only 4 of the uninformed innocent were deemed truthful there would be some meaningful conclusions to draw from such an instance. Collapsing the two pools into one innocent pool would produce 14 out of 20 correct calls or 70% . On the other hand if we reverse the numbers I previously mentioned and it was found that none of the informed innocent used countermeasures the same results could be construed as to the 70% but there may be some inference as to the adverse effect of knowledge of the procedure on an innocent examinee.

posted 06-02-2004 02:56 PM            

In a DLC test, one is not simply told to lie to a particular question as some might think. Instead the examinee is told (after a proper DLC introduction) to think of a particular incident and to be sure he pays attention to how his body feels so that he responds "appropriately." Just what does that mean? Who knows? But, in theory it is enough to shift the truthful persons attention to the DLCs. (And there is a significant amount of data to support that assertion. The jury still seems to be out on criminal tests and DLCs, but the TES research looks pretty good for screening exams; though, I don't know how well that would apply to what we do for reasons I won't bore you with here.)

I've run a few DLC tests, including the TES format as a screening exam. I'm not a big fan, but I have had some (limited) success with them.

One of the big downsides, as I see it, is the CQs are so much easier to recognize. After all, you tell the person what they are. (I'll post below how to set them up for those who are interested.) A George disciple would have to be a dunce not to know where to try his "techniques."

I'd rather use a PLCQ test with disguised CQs than a DLCQ test for that reason. Plus they just "feel" better. (Not very scientific though, huh?)

-----

I like the R/I as a screening technique, but not as a diagnostic test. I don't think twenty-five consistent charts necessarily prove anything in regard to deception. After all, you might just end up repeating your error(s) twenty-five times. I use a successive-hurdles approach so that I can make a decision of DI based on a diagnostic test that is less inclined to result in a false positive. (By the way, there are many pro-polygraph folks who think the R/I is worthless because the examinee could fear detection or (fear) being branded a liar when he is in fact truthful.)

----

Here's the DLC info I mentioned above:

• Explain that you need a sample of what it looks like when subject tells a lie.
• Ask if the subject has committed an offense that most everybody has committed, e.g., lying. (He should not tell you what he did.)
• Once the subject admits to having committed an offense, tell him to answer the question with a “No” during the exam. In addition, tell him to think about an occasion when he committed the specific offense, i.e., if the question is about lying, then he should think about a time he told lie, and more specifically, one for which he still feels guilty.
• Inform the subject you want him to think about how he feels when he lies because how he responds is critical for the outcome of the test. If he doesn’t respond appropriately, the test will be inconclusive and have to be repeated at a later date.

posted 06-07-2004 11:51 AM            

What are the probabilities that a person on 4 or 5 charts will react consistently and significantly to the same relevant question, 75-80% of the time simply out of fear they will be falsely branded a liar? I don't know, but I would assume pretty low.

I'm not ready to throw the towel in on the R and I. I still think we need more (new) research.

Jim

P.S. My usage of DLC's tend to work just a good as standard CQ.

posted 06-08-2004 11:15 AM            

I started using them about 4 years ago primarily with IAU testing after I noticed almost every officer entered the suite with knowledge of the CQ technique (George's site conveniently provided to and endorsed by the local PO Assoc).

Also, they believed anything they "said" in the interview could be held against them, regardless of whether it pertained to the incident at hand, many were "balking" at answering or even discussing standard CQ's during the preview of test Q's. (I initially thought this was good, but they were refusing to answer and it was becoming argumentative...)

I started DLC's as a way of handling their reaction and rejection and having them silently answer "yes" (shaking head) rather than aloud, then explaining the rest of the DLC prepatory. It works fine.

BTW, to answer your question, most IAU stuff is single issue, Bi-Zone Series.

Jim

posted 06-08-2004 11:23 PM            

I think that the R&I is a good substitute for those instances when there are extenuating circumstances but it falls short in my opinion when it comes to a solid evidentiary test. For those who are mechanically inclined, I liken the R&I to a crescent wrench. The crescent wrench will get the job done for you most of the time when you do not have the right tool but the times it does not it will round the nut. There is a lack of control and consistency and an over abundance of subjectivity in the R&I. What one examiner may consider significant another may not, a loss of reliability that we have at least gained with the use of PLCT. There are also an overt amount of other variables that can come into play and cause a reduced response. If we are to forward the profession into the evidentiary stage, we need to evaluate the procedures we use and insure that they will stand up to the current rules of evidence. I know that this is the last thing on some minds but, as I have said before, when a procedures liability outweighs its utility it is doomed.

posted 06-09-2004 11:27 AM            

I am hopeful we will be there someday, so yes, it is always on my mind. I think the idea of psychological set is one of those things in our way. It is never (and I've done a number of searches) cited in any of the psychological literature. (I understand Jim Matte found such a reference in some obscure 1940s (ish) publication, but that is hardly a solid basis for such an assertion.)

I'm not saying the whole idea is without merit. I think science has shown liars react more strongly to RQs than CQs, and innocent people do just the opposite. We have used the scientific method many times over and found that to be true (though not 100% of the time, but what test is?).

In other words, one type of qustion is more salient than the other, and that is what we are measuring. Why do we continue to propose theories (or use terminology) those with PhDs in psychology and psychophysiology have never heard of - or will testify are not supported by science - when we don't have to?

It seems we are our own worst enemies.

Someone asked if a person would continue to react in an R/I test out of fear of disbelief. There is no scientific reason to believe a person would stop reacting any sooner than a person who is being deceptive.

Let me restate my position though: I like the R/I in the situaltions I mentioned before (and some others, too).

posted 10-23-2012 10:04 PM            

Corey

posted 10-24-2012 08:51 AM         

quote:

---

And the test worked on all of us when we tested each other in polygraph school, fully aware of what was going on. Imagine that.

---

Barry,

How many of your class members were employing CMs as if their lives depended on it?

posted 10-24-2012 03:58 PM         

quote:

---

How do you know that matters?

---

C'mon, Barry. Everyone knows it matters.

Knowledge of how polygraph works and CMs are essentially joined at the hip.

That's why the Canadians, in their how-to literature, breathlessly caution against letting the "secret" of the CQ's purpose out of the bag.

Work with me for a moment and imagine this scenario...
An applicant shows up for a LEPET test. The jocular police examiner asks the applicant what he knows about polygraph. The applicant says:

"What do I know about the polygraph? Everything. I've read TLBTLD seven or eight times. That book spooked me, so I took three practice LEPET tests with a private examiner who's a retired FBI agent. That guy really put me through the meatgrinder. Bill Iacano is my godfather, and I was an usher at Jeffrey Rosky's wedding a few months ago. Jeff and I go way back. By the way, I keep a copy of TLBTLD handy as a bathroom reader, because the polygraph is a bunch of crap. Everyone knows it, except for the polygraph industry and those with a vested interest in the so-called test. The whole thing is bogus. Hell, the American Psychological Association, the American Medical Association, and even the Supreme Court of the United States all say the test is fundamentally flawed and unreliable."

Question:
Would the examiner have a negative bias towards the applicant?

I say yes, he would, because knowledge of the test is associated with CMs.

Dan

posted 10-24-2012 05:41 PM            

I presume every candidate has read up on polygraph. I don't care what he knows.

Just because the Canadians believe something doesn't make it true. Nor is something true just because a court says it. Iacono's testimony was used to admit polygraph in Alaska last week.

posted 10-24-2012 06:15 PM            

quote:

---

Barry,

How many of your class members were employing CMs as if their lives depended on it?

Dan

---

"lives depended on it," is waaay too dramatic. Probably nearly nobody's life actually depends on a polygraph.

You are correct that it would be great to have field studies alongside the laboratory studies. In all likelihood those field studies - when they are done - are going to be in the form of uncontrolled surveys and not double-blind controlled experiments. Why? Because live field studies have limitations in terms of what we can actually control.

Bill2E

quote:

---

Psychological set is important. I am aware that we are trying to find differently in all of the current research. We have to be careful when we say it is untrue and there is another reason for reactions. The "theory's" we put out there are all theories not facts. I'm waiting for a true study that clearly demonstrates Psychological set is not correct. The DLC testing can also be attributed to Psy Set!!.

---

Interesting perspective. It depends in how loosely or rigidly you use the term "psychological set." I'm OK with using it loosely - because it is a term of convenience and not a psychological construct - in which case I agree that it is still a useful way to caption the fact that attention and responses get loaded onto more important stimuli. Why are those stimuli more important? A rigid application of the psyc-set hypothesis says this: fear. That is the only problem - the defined psyc-set hypothesis is limited to emotion and fear alone, and cannot explain why the reactions of truthful persons load onto questions to which they are told to lie and answer no. The psyc-set hypothesis, in its narrow form, also falls short of explaining why the polygraph works with psychopaths. Psyc-set (fear loading) would have us expect that psychopaths either pass or produce inconclusive test results. But the evidence says the test works.

Try this: a recent class of 13 students completed a SPOT exercise in which they embed the name of their first girlfriend/boyfriend in a list of random names and then answer the series of questions 3 times - forward, backward, and forward again - while repeating each name (not answering 'yes' or 'no' and therefore not lying). There are no consequences for failing and no rewards for passing. 11 of 13 student examiners correctly determined the first girlfriend/boyfriend of the examinee. For some psychological reason the examinees' physiological responses loaded onto the names of the first girlfriend/boyfriend. Why? Fear? Probably not. Could be emotion not limited to fear. Could be memory or mental effort (fantasy???) Could be nothing more complicated than simply behavioral conditioning based on past experience (first holding hands, first kiss, first hormone rush... whatever...).

Physiological reactions during SPOT are of the exact same observable and measurable type of reactions observed during CQT testing? Why? Answer: because the underlying psychological mechanisms might actually be somewhat similar regardless of the type of PDD test (CQT/CIT/RI/PLC/DLC).

Most likely the reason that attention and physiological responses load onto certain questions is some combination of all three: emotion, cognition, and behaviorally conditioned response. That is the only difference between the psychological construct of "salience" and the polygraph term "psychological set." Want to keep using the term "psych-set." No problem. Just don't limit your definition and conceptual understanding to emotion and fear alone - include emotion, cognition and behavioral conditioning in your understanding of "psych-set" and you are good to go. Obviously some people are more emotional than others, some less, some more cerebral, and some people probably don't stop to think much or engage in introspective reflection on their subjective emotional experiences. We may not need to know the exact proportions of these three basic forms of psychology - just know they all matter to some greater or lesser degree.

So, studies are not likely to say that "psych-set" is wrong. They are likely to say that it is either incomplete, or it is not well defined if we define it rigidly. If we define it broadly, then it becomes essentially the same construct as salience, with no fixed assumptions about the exact proportions of cognition, emotion (or which emotion) or conditioned response.

Dan:

quote:

---

Question:

Would the examiner have a negative bias towards the applicant?

I say yes, he would, because knowledge of the test is associated with CMs.

Dan

---

OK. What is your solution? Some would say that a solution would be to further structure and standardize the test administration and test interpretation so as to reduce the influence of that their bias.

Personally I believe we are naive to assume that any examinee has not read about the polygraph.

Dan:

quote:

---

Knowledge of how polygraph works and CMs are essentially joined at the hip.

---

Why? I imagine there are a lot of federal employees who have to occassionally take polygraphs while possessing a large or small amount of information about the polygraphs. If knowledge and CMs are inseparable (i.e., joined-at-hip metaphor) then they would all be guilty of using CMs every time... Yet this is probably not the case.

more later - i'ma go get me some new clothes now that my luggage is lost again.

.02

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 10-24-2012 08:52 PM            

See http://www.radford.edu/~jsking/Orienting%20Response.doc

posted 10-25-2012 08:20 AM         

quote:

---

I could write a book on logical fallacies just using your writings.

---

What does polygraph have to do with logic? The very weak explanation of how polygraph "works" defies logic.

quote:

---

Nor is something true just because a court says it.

---

In writing the majority opinion for the Scheffer case, Justice Thomas said "...even the best of exams are fraught with uncertainties." Do you disagree with that?

quote:

---

David Raskin asks the following question: Why do guilty people show up?

---

Because they are tricked, cajoled, pressured or otherwise "forced" into taking the the test.

Dan

posted 10-25-2012 09:14 AM            

quote:

---

Barry:
quote:
I could write a book on logical fallacies just using your writings.
What does polygraph have to do with logic? The very weak explanation of how polygraph "works" defies logic.

---

posted 10-25-2012 10:00 AM         

In your opinion, is the issue of knowledge having an effect on the polygraph "settled"?

In other words, a subject having advance knowledge of how polygraph works, knowing ahead of time the precise relevant questions, and even having in-depth knowledge of CMs beforehand simply doesn't matter. Is that what you are saying?

I'm asking because so I can better defend my open-book approach when it gets criticized.

Lawyers and PDs here in NH are already bristling at my approach, which I attempt to explain is not only sound, but destined to eventually become "the new normal."

It would be an enormous help if the president of the APA officially shared my belief.

Dan

posted 10-25-2012 01:06 PM         

One test doesn't prove a thing.

I'm simply asking the APA president to clarify for me his position -- at least as I understand it from his postings on this forum -- on whether advance knowledge matters in polygraph tests.

The cops are downright befuddled when I broach this concept.

If I have buy-in from Barry, the problem goes away. Why? Because I don't have to "prove" a thing to support my position, as the word from the top is consistent with mine.

Dan

P.S. I'm crossing threads here, but go to minute 24:00 of the video linked below and watch the acquaintance test. Then tell us, based on the tracings, why Brian was justified in choosing number 4 as the key.
http://lawmedia.lclark.edu/LawMedia/SilverlightPlayer/Default.aspx?peid=6f5b20932158401ba36cd1b2854f2fa01d

posted 10-25-2012 01:48 PM         

My point still applies.

Later in the video, Brian explains that the channel tracings are weighted (for scoring), with EDA getting the lion's share.

The EDA on number 6 is almost flat, but the EDA rise for number 8 looks timely and significant.

I heard his explanation of the pneumo tracing associated with number 6. But...

posted 10-25-2012 04:49 PM            

Honts, C. R. & Alloway, W. (2007). Information does not affect the validity of a comparison question test. Legal And Criminological Psychology, 12, 311-312.

Go here for abstracted information: http://scholarworks.boisestate.edu/psych_facpubs/52/

While the study agrees with your position, you might be prepared for some of the caveats that would go with this paper. First, it is only a first look and there are no other studies. Whether the Honts et al results are a statistical fluke or heaven's honest truth would need replication to determine. Second, the sample sizes for each cell were really small, only 10 each. Frankly, it would have been more surprising had Honts found a difference. To have an 80% chance of detecting even a 20% difference would take at least 50 samples per cell. Smaller differences need even bigger samples to detect them.

I can't speak for Barry, but until we have more studies, it's taking a sizeable leap to say there is no risk with the open book approach. A topic worthy of debate, though, as I think this holds the possibility of being an example of us collectively assuming we knew something that might, in the long run, turn out to have been wrong. Time will tell.

Something occurs to me, though. If the examinee knows what PLCs are for, in your experience is there a tendency for PLCs to in effect become DLCs?

posted 10-25-2012 05:19 PM         

quote:

---

Something occurs to me, though. If the examinee knows what PLCs are for, in your experience is there a tendency for PLCs to in effect become DLCs?

---

Interesting angle. Never thought of it that way. But, I'd have to say yes -- especially for PCSOT subjects who have been around the block a couple of times -- some "experienced" test takers offer absolutely no resistance to the PLCs.

However, unless I'm running a pure DLCQ, I make no scoring adjustments to the pneumo channel.

Dan

posted 10-25-2012 09:48 PM         

quote:

---

I presume every candidate has read up on polygraph. I don't care what he knows.

---

Don on 10/25:

quote:

---

I can't speak for Barry, but until we have more studies, it's taking a sizeable leap to say there is no risk with the open book approach.

---

It's fascinating to see this issue being walked back.

Barry's silence is deafening.

I wonder who reeled him in -- and why.

posted 10-26-2012 03:21 AM            

quote:

---

In your opinion, is the issue of knowledge having an effect on the polygraph "settled"?

---

Chiming in here... I know you asked Barry...

It is a kind of underlying principle in science that nothing is ever completely "settled." This is because we can never know everything. We have our expert opinions - called "hypotheses" - and most of them turn out to be wrong when we actually study them. So we have to be diligent and focus on the proven things in order to develop or improve any new hypothesis that will work.

Hypothesis become theories when there is replicated evidence to support a hypothesis. Theories, therefore, are scientific ideas for which there is evidence of support. There is often a mix of evidence in support of and evidence against any hypothesis or theory - and no form of evidence is perfect. The ultimate head-banger is this: all theories are wrong. They are wrong in that it is impossible to know everything that all theories are therefore incomplete. Our task is to continue to expand our knowledge by continuing to evaluate our theories by applying them to new and different phenomena and new and different ways of testing their limits. When we find some phenomena that a theory cannot account for, then we are obligated to modify something. Hint: its not OK to modify or ignore the evidence. We upgradde the hypothesis (with a double dose of pimpin'). So - our ideas start life as hypothesis, are regarded as theories for a while, and then are regressed to the status of hypothesis again when we find new evidence and an improved hypothesis that forms a new theory.

Another underlying principle in science is that a single study is worth almost nothing without replication. Yet another principle is that everything - everything - is ultimately a probability, and all results are probability statements. No matter how much confidence, charisma, and bravado we add to the message that someone has failed (miserably) or passed (with flying colors) the underlying message is always probably lying or probably telling the truth.

So...

Nothing is ever really "settled."

This discussion exists in the space in between long-held beliefs and experience about the need and importance for secrecy around polygraph procedures, and an absence of evidence that secrecy or openness make the kind of difference that we would have assumed. Polygraph continues to work at the same level or better in the age of internet information. The test may be different, but not DOA.

Keep in mind that our profession, including police examiners, government people, and pcsot examiners, sometimes work quite close to the edge or boundary between safety and catastrophe. We will be wisely reluctant to make furtive changes for which the outcome will seem to be or will be uncertain. At the present time we know what to expect. Unmeasured change can bring unmeasured results. Any expectation that people immediately and comfortably endorse an idea that unsettles our sense of safety may be an exercise in frustration.

I suggest you proceed cautiously - remembering that the book may already be no longer closed.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 10-26-2012 07:36 AM         

Won't passing "scientific muster" -- and acceptance by the community -- ultimately hinge on polygraph's viability in an open-book world?

Efforts to keep the book closed constitutes just another form of us whistling past the graveyard.

If the test is compromised by having knowledge of how it "works," then what does that mean for the future of polygraph in the Information Age? Further, how will more "certifications" advance polygraph if the same old criticisms are not debunked?

posted 10-26-2012 10:53 AM            

As Ray said, nothing is ever settled in science. We just update our theories as more evidence becomes available. Additionally, it's another logical fallacy to appeal to authority, so don't take my word for it: look at the evidence and evaluate appropriately.

Ray also cautions us to be careful of the CM studies. I agree, but they are still instructive. Don pointed out Charles Honts's work, but before that was Lou Rovner's. He had an "information" group and an "information plus practice" group. Thus, one of the groups was very similar to the Honts' participants. He too had small cell sizes (but larger than the more recent study).

Honts and Alloway didn't use an INC zone, so I combined those results with Rovner's because he reported with and without an INC zone. The information group average accuracy (combined as in a meta-analysis) was 78.9% (62.7 to 89.2% at 95% confidence level) and the information group was 82.1% (66.7% to 91.3%), which is not statistically different.

Now, is so called open book testing a good idea? I don't know. Honts' cautions that truthful people who use CMs - something also described and encouraged in the book you have open - increase their chances of failing. So, knowledge of the test combined with CM instructions might be harmful to truthful examinees - not because knowledge of the test structure is a problem, but rather because of the risk of being persuaded to use CMs. I choose to talk about CMs and tell people why they wouldn't want to buy that nonsense.

posted 10-26-2012 08:27 PM            

I know where it could lead.........insert dramatic pause here...........

Yep, you guessed it. Those familiar with Thomas Kuhn (probably most of you!) read no further. Or do read and be re-inspired of the possibilities:

"Kuhn states that scientists spend most (if not all) of their careers in a process of puzzle-solving. Their puzzle-solving is pursued with great tenacity, because the previous successes of the established paradigm tend to generate great confidence that the approach being taken guarantees that a solution to the puzzle exists, even though it may be very hard to find. Kuhn calls this process normal science.As a paradigm is stretched to its limits, anomalies — failures of the current paradigm to take into account observed phenomena — accumulate. Their significance is judged by the practitioners of the discipline. Some anomalies may be dismissed as errors in observation, others as merely requiring small adjustments to the current paradigm that will be clarified in due course. Some anomalies resolve themselves spontaneously, having increased the available depth of insight along the way. But no matter how great or numerous the anomalies that persist, Kuhn observes, the practicing scientists will not lose faith in the established paradigm for as long as no credible alternative is available; to lose faith in the solubility of the problems would in effect mean ceasing to be a scientist.In any community of scientists, Kuhn states, there are some individuals who are bolder than most. These scientists, judging that a crisis exists, embark on what Thomas Kuhn calls revolutionary science, exploring alternatives to long-held, obvious-seeming assumptions. Occasionally this generates a rival to the established framework of thought. The new candidate paradigm will appear to be accompanied by numerous anomalies, partly because it is still so new and incomplete. The majority of the scientific community will oppose any conceptual change, and, Kuhn emphasizes, so they should. To fulfill its potential, a scientific community needs to contain both individuals who are bold and individuals who are conservative. There are many examples in the history of science in which confidence in the established frame of thought was eventually vindicated. Whether the anomalies of a candidate for a new paradigm will be resolvable is almost impossible to predict. Those scientists who possess an exceptional ability to recognize a theory's potential will be the first whose preference is likely to shift in favour of the challenging paradigm. There typically follows a period in which there are adherents of both paradigms. In time, if the challenging paradigm is solidified and unified, it will replace the old paradigm, and a paradigm shift will have occurred."

The Structure of Scientific Revolutions / Wikipedia

posted 10-26-2012 10:30 PM         

You're pretty smart for an Okie.

Dan